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to XKEYSCORE 
^Purpose and Capabilities 




XKEYSCORE Databases 
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■ XKEYSCORE performs filtering and selection 
to enable analysts to quickly find information 
they need based on what they already know. 



■ XKEYSCORE also performs SIGDEV 
functions such as target development to allow 
analysts to discover new sources of 
information. 
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■ XKEYSCORE processes data at field sites, 
where it is collected, and allows analysts from 
all over the world to query it. 



■ At field sites, the XKEYSCORE software can 
run in clusters of few or many servers, giving it 
the ability to scale in both processing power 
and storage. 

■ All processing is plugin or fingerprint based, 
which allows new capabilities to be quickly 
deployed to support operational needs. 
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■ XKEYSCORE is a Computer to Computer 
(C2C) exploitation system. 

■ It is a fully distributed processing and query 
sy ste m . 

■ XKEYSCORE can run on multiple servers. 

■ Plugin and fingerprint architecture allows new 
capabilities to be quickly deployed. 
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XKEYSCQRE is typically installed with Red Hat 
AS5u8 operating system. The suggested disk set up 
is: 



• Set up separate partitions for / (root), /var, 

/tmp, and I export/data 

XKEYSCORE clusters can be composed of three 
different functionalities, which are: 

• One host acts as the web server/user interface, etc... 

• Another host normally runs as the real-time processing unit 

• Other host acts as the search or query system. 

Hybrid system can perform multiple roles on one 
server, which enables efficient registration. 

• process_data_parent 
o 1 queryjproc 
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The backend is where the raw data for 
XKEYSCORE is processed; that is, we 
receive information from our sources fe.g. 
WEALTHYCLUSTER2), process it, and store 
it into a database. 



[sessions] 



-> [processing engine] 



(database) *€ ■> (user queries) 
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Data Flow - XKS Cluster 
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■ A cluster is comprised of one master server 
and one or more slaves. 



■ All slaves in a cluster have their own copy of 
configurations 7opt/xkeyscore/config) files via 
the r@yn© pusG^omfig cronjob. 
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" .'here are two types of databases on art 
XKEYSCORE system: insert (iO) and query 
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iO 








qO 



NOTE: sotf_input_proc is now called, sotf_dist 

process_dataN’s are now called, process_data_parent 
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_input_proc and sotf_dist take in sessions 
from the front-end and load balances them 
across multiple process_data_parent’s. 



■ process_data_parent is responsible for 
processing sessions and extracting metadata 



■ xks_meta_ingester takes the metadata from 
the process_data = parent’s and writes it to the 
insert database, iO 



■ register_metadata_tables takes completed 
insert tables, indexes them, and moves them 
to the query database, qO 
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Operating System Services 



VHTTPD 




IVlount Points 

/xks_data 

Directory Structure 
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■ XKEYSCORE is typically installed on servers 
running Red Hat Su8 operating system. 

■ This section discusses common operating 
system services used during XKEYSCORE 
operation. 
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The rtrays^l daemon is a SQL-based database 
server for processing, querying, and is needed 
for ■ XKEYSCORE GUI. 



It is required on all servers for administration, 
processing, and querying metadata in 
databases. 
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■ [Mounting a directory uses the N FS service. 

■ N S allows file systems that physically reside 
on one computer to be shared by other 
computers on the network. 

■ The machine with the hardware containing the 
directory must allow the hardware to be made 
available to other machines. 

■ Required on all computers for clustering. 
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■ /etc/exports 

• /export/data/xkeyscore master(rw:; slav6(rw; 

• /opt/xkayscore/config/loadserver *(rw) 
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■ Computers requiring shared access to the 
/export/data/xkeyscore directory must be told 
where to find the directory. 



• This is accomplished via automounting. 

■ The motets daemon listens for computers 
trying to connect to the directories, or mounts, 
that it is responsible for. 

■ The mounts are dropped after a time out, but 
aut®fs remounts the drive when drives need 
to be accessed. 
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■ For a clustered XKEYSCORE, automounts 
must be set up on all of the computers in the 
cluster. 



■ auto. master and auto.data files in the /etc 
directory must be edited or created. 

■ When finished, the mounted directories on the 
remote machines can be accessed. 

■ The oper account should have full read/write 
permissions on all shared drives. 
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■ auto.msisttir- designates mount points on the local 
computer and the directory to mount on the remote 
server. 



• Example: 

► /xks_data /etc/auto, data — timeout=6G 

■ giuto.datii - enables all servers to see the 
/©xport/data/xkeyscore directory on other machines 
and locate databases, archived, data, and 
MAILORDER directory. 

• Example: 

► xksl -rw,sofUntr,tcp xks1:/exponl/data/xkeyscore 

► xks2 -rw,soft,intr,tcp xks2 : /ex port/data/x keys core 
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/®ptek@^S(g©r ©I - contains all of the 
XKEYSCORE software. Software includes 
the GUI, processing, scripts, and 
configurations. 

EUrfl - XKEYSCORE environment variables 



• bmmnl - contains the beacon perl script 
fshm_beacon.pl) and a link to the beacon 
configuration file (shm_beacon.config). 

• bin.@h©lls/and fein.shells/s^sadmin - contains 
miscellaneous bash, python, and C shell scripts. 

o [byil©]/ - contains libraries and plug-ins. 

• install/ - contains installation scripts. 
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/®pTOk®fi(gor®/©onfig]/ - consists of sub-directories 
and each contain configuration files for building and 
running XKEYSCORE. 

• cronftafe/ - contains the master and slave crontab file. 



• dlietionarigs/ - contains the dictionary files for the filtering, 
selection, TRAPFICTHIEF, CADENCE, fist tables, and any 
other local dictionaries. 



• mise/ - contains miscellaneous per-plug-in configuration 
files, (i.e. sotf_input_proc.xml ). 

• plugins/ - contains event handler configuration files for 
each of the plugins (default.xml). 

• www/- contains web configuration files and xscore.cfg. 

o SERVICE/ - contains the config files for all the services 
needed by XKEYSCORE (httpd, php, mysqld, etc.) 
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/®^p@rt/(dlii'iiiite[k(iys©<0)r(i/ - is used for both internal 
databases and metadata archive databases, input, 
output, and archiving of data. 



• ar©hi¥@s/ - (optional) destination for processed content 

• inputs/ - (optional) used for file based input 

• mysegl/ - location of the MySQL database consisting of 
admin, insert, and query databases. 

• outputs/ - (optional) contain the following sub-directories: 

► manlordler/ - pickup point 

► mail@rd®r_w©rCting/ - file creation point before being moved to 
mailorder/ 
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■ M^sjskta/ - logical mount point for all other 
XKEYSCORE (including itself) 
/export/data/xkeyscore. 

• - mount point for the hostname's 

local directory /export/data/xkeyscore (referenced 
by host name). 

> All servers must export their /export/data/xkeyscore 
directory and mount this on the /<hostname> directory 
for each hostname of each machine, including itself. 
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Accessing the GUI 
V Exiting a Session 
Main Menu Bar 
MyXKS 

V Admin 

Computer Resources Option 
Start and Stop Processing 
v un a Process Manually 
Users 
Search 

Workflow Central 
Results 

V Fingerprints 
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■ In the address field of a web browser, type 
https ://<m aster hostname or IP address>. 

■ PKI's or a UserlD and password are required. 
After successfully launching a new session, 
the XKEYSCORE WELCOME window 
appears. 

• Mote: Compatible web browsers for XKEYSCORE 
version 1.5 are: 

► Internet Explorer is not supported 

> Firefox/3.0.* and above 
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I OPTION 

Home 


£gr_jm 

DESCRIPTION 

Returns to the main page. 


RflyXKS 


Can edit user settings, disable/enable access to 
databases, edit a search form search setting, and restore 
default settings. 


Mmin 


Computer resources, Input Directories, Category Throttle, 
Search DBs, and DB Registration settings. 


Users 


Contains User Accounts, Clearances, Privileges, Send 
Email, Users Online, My Auditees, My Audit Logs, and All 
Audit Logs. 


Search 


Provides different search query forms, such as email 
addresses, category, full log, and user activity. 


Workflow Central 


Request , modify, and view standing queries that will 
execute at a specified time or interval. 


Results 


Can search personal searches by date time, query type, 
query name, output table, and user. 


Fingerprints 


ingerprint builder and viewer. 


[feflgip 


Brings up Google Earth 


HeDp 


Help Documentation, XK Forum, Account Maintenance, 
and About XKEYSCORE 



US, CAN, GBR, NZL 
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SyB-MENU 

©ption 


DESCRIPTION 


Computer Resources 


Allows for process configuration and management. 


Input Directories 


Contains the configuration for file-based input directories. 


Category Throttle 


Edit CADENCE quota limits by category and/or fist table. 


Search DBs 


Configuration for query databases which are queried 
when a search is submitted. 


DB Registration 


Contains the mapping from insert databases to query 
database. 


News 


Add, modify, delete mandatory and home page News. 
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■ The P rp.dplls i n}q'4> O ojjfap fe r 
option from the ADMHM menu allows control of 
the entire daemon-styled, or continuously 
running, processes for XKEYSCORE. 

Processes appears in a table following the 
convention: 



<PROC_HOST><PROGRAM_NAME><PROGRAM_ARGUMENTS> 
xkey0 1 process_da ia_parent 
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Navigation Flter x |EF| 

E0 Preeessrg 

^ Computer Resources 
i~3 Input Directories 
i=r] Category Throttle 
1-63 Databases 
"-] Search DBs 
l=3 DE Registration 
t £j Utilities 

(=3 Casenotati&n Blacklist 
1=3 Reload Corfig Files 
i=3 Nswe 

®3, Ip Summary Table 
C'ashlogger 
(rn Startup 
'A Query Profiler 



Color Key 










STOPPED 


STOPPING 


STARTING 


RUNNING 


WONT START UP 


STOPPED? 

APR LAUNCHER STOPPED 


STOPPING? 

APP LAUNCHER STOPPED 


STARTING? 

APR LAUNCHER STOPPED 


RUNNING? 

APP LAUNCHER STOPPED 


WONT START UP? 
APP LAUNCHER STOPPED 



Computer Resources 


«! Help 


Add Actions T 


AppLauhchei h Running 














Aed&ns 


Prcc Hesi 


Program Marne 


P/ogre^ Arguimertlt 


Program p£> 


CoiEunefttted $1; fit vi 


Sixtus 


DeteSme Started 


Detetime Stepped 


• 


tlxksvrOI 


OUld 




7S07 


RUM 


RUN 


201 2-1 2-03 1 &: 44; 49,0 


201 2-12-0315:44:40,0 


• 


tlxksvrOI 


query _proc 




309SS 


RUM 


RUN 


201 2-1 1-27 17: 33 57.0 


2012-11-2717:32:47.0 


• 


tlxksvrOI 


Aheckjmailorder_sre.pbp 




31019 


RUM 


RUN 


2012-11-27 17:3357.0 


2012-11-2717:32:47.0 


• 


tlxksvrOI 


:<ks_metajnciester 




31051 


FUN 


RUN 


201 2-1 1 -27 1 7:33 57 ,0 


2012-11-2717:32:47,0 


• 


tlxksvrOI 


tiicKstreamservJcesJi 




31053 


PUN 


RUN 


2012-11 -27 17: 33 57.0 


201 2-11-2717:32:47,0 


• 


tlxksvrOI 


query _dispatch 




1311 


FUN 


RUN 


2012-12-00 21:0303.0 


2012-12-03 21:03:07,0 


• 


tlxksvrOI 


f.ileJnputj?roe 




311CWI 


RUN 


RUN 


2015-11-2717:3357,0 


301 2-1 1 -27 1 7; 33:47,0 


• 


tlxksvrOI 


Xks„system_monitc:r 




i zm 


RUN 


RUN 


2012-11-27 21:01:13.0 


2012-11-27 21:01:13,0 


• 


tlxksvrOI 


sctftodl 24server 




mm 


FUN 


RUN 


201 2-11 -27 1 7:33 57 .0 


201 2-11-2717:32:47,0 


• 


tlxksvrOI 


lomeart.sb 




31124 


FUN 


RUN 


201 2-11 '27 1 7:33 57 ,0 


201 2-11-2717:32:47,0 


• 


tlxksvrOI 


cadence Jasklngjcroc 


-myfdiXYD --pddg IE -digraph X5 


31136 


FUN 


RUN 


2012-11-27 17:3357.0 


201 2-11-2717:32:47,0 


A 


tlxksvrOI 


xks^server^stats 




31138 


FUN 


RUN 


201 24 1 -27 1 7:33:57 .0 


201 2-11-2717:32:47.0 


• 


tlxksvrOI 


mailorder _p.rec 


-oopydir /exp grtfdataAd<£Y3OTefoJt . .. 


31110 


FUN 


RUN 


201 2-11 -27 17:3357,0 


2012-11-2717:32:47.0 


• 


tlxhsvrOI 


regista _iinctadata Jablcs 


-loglevcl cmx 


31143 


FUN 


RUN 


2012-11-2? 17:0357.0 


2012-1 1 -27 1 7:03:50.0 
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■ The 2&@ = ap>pJ®un©h@r process runs on all 
servers from the inittab. 

■ It tells the computer which program to run by 
looking at its tasking host. 

■ / opt/xkeyscore/config/www/xscore . cfg 

• The config file specifying the location of the 
tasking database. 

■ Processes can be stopped, started, edited, or 
deleted from the Computer Resources 
window. 
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■ Add a new process - click Add 

■ Edit a process - click Step in the ACTION 

column, then click Edit. 

■ Delet© process - click Stop in the ACTION 
column, then click Delete. 

■ Stop the App Launcher - disables the 
xks_appjauncher on every host. 
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■ Visual cues in the form of colors are used to 
help identify activities performed by 
XKEYSCORE and seme as status indicators 
for monitoring purposes. 

• ©®d - indicates processes have been stopped 

• ©r@@n - indicates processes are running 

• Y®ll®w- indicates processes are starting 

• ©rang© - indicates processes are being stopped 

• Whit® - indicates processes won’t start 

■ Visual cues are also available in the 
COMMANDED STATUS and STATUS 
columns of the table. 
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Resources-Start/Stop Processing 

HOfOioooi | 

o 



ir l 



W 



liSrnm 



mn 






It may be necessary to stop or start processes 
for troubleshooting or for a graceful server 
restart. 



■ Mivisrtoal processes aondl pregrams - Click 
Step in the HCTloH column. To start it, dick 

Run. 



■ To stop all individual programs, select 
zd©TI©dS-^Start/St®p lrt®s®yrs@©. Enter the 
program name in "IMOOldMtlS field, them click 

©d 

■ Can use odes pr@© actions and commands to 
do the same function 
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■ All Processing - select START/STOP 
R@s®urc@s from the ACTIONS drop-down 
menu, leave the PROSHffts and ON 
HO©' = S fields to their defaults, click OCA 

■ Specifying programs or h®sts - select 
STOP or S c |j enter a wildcard expression 
such as i or ! in the '^AOSCdAMIS or HO: 
field, and click OCA 

• Example: process* 

■ Alternatively, in a terminal window can run: 

o xks proc stop process* 



TOP SECRET //SI II REL TO USA, AUS, CAN, GBR, NZL 



40 





TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL 



■ It may be necessary to run a process manually for 
troubleshooting purposes. To run a process 
manually: 

1 . Launch the GUI and log on as oper or admin. 

2. Click ADMIR » Processing > Computer [Resources 

3. Click Stop in the ACTIOM column for the process. 

4. Open a terminal window and ssh to the host running the 
process, as the user 'oper'. 

5. "ype ps -ef | grep <process name> to verify 

that the process is stopped. 

6. Type <program nameXprogram argument> 

■ Example: 

query proc <program arguments , if any> 

--loglevel debug 
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nr Home 



MyMG >4 Admin X 



=3 Ussr Accounts 
=?1 Clearances 



£ 1 Privileges 



El 5end Email 
Liters Online 



I i 



<5 



Navigation “Iter * 






This menu is only accessible to users with 
system administration privileges. 

An SAcan add/modify user accounts, add 
groups, clearance levels, privileges, and 
email users from this menu. 
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From the main menu bar, click ipMS to view 

your profile, accesses, privileges, auditors, 
settings, fingerprints, workflows, and recent 
results. 



■ Right click on any search form name to add a 
shortcut for that search form. 





x x i — — : 




Navigation Filter * ,[£l itr| 


» 1 > u t ^ 


St 


m Full Log DNI 


\ a n l=i 


i 1 


HTTP Activity 


— ^ — T 

Full Log DNI HTTP Activity My Fingerprints My Workflows My Recent 


Profile 


S My Fingerprints 


Results 




1^1 My Workflows 






§ My Recent Results 






l£| Pro Pile 
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From th© main menu bar, click SEARCH. 
Menu options display in the vertical pane on 
the left. 
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a ti on Fil ter 

s-~1 Search Wizard 
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'■ ^ Registry 
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m User Assist 
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■ When choosing a plugin type from the menu 
options, the only data searched is the data that 
was identified as a hit when the plugin was 
processed. 



SUB-MENU 

OPTION 


DESCRIPTION 


Category DNI 


Searches dictionary category hits. 


Full Log DNI 


Searches all sessions received by XKEYSCORE. 


User Activity 


Enables a user to search by a user’s activity. 
Example: a user can find a hotmail user’s 
msnMailToken 
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■ All searches are conducted on database 
tables where the results of the XKEYSCORE 
engine are stored. 

■ Each row of a database table contains values 
from an individual session that was identified 
as a hit by XKEYSCORE when that plugin or 
microplugin processed the session. 

■ Each search type query is related to a plugin 
or microplugin, which performs the metadata 
extraction. 



TOP SECRET II $\ It REL TO USA, AUS, CAN, GBR, NZL 



46 




N, GBR, NZL 





■ Search details can be accessed from the 
Search status window by clicking Details. 



■ CUAREOT SEARCH DETAILS window 



displays and allows the user to watch a query 
run through the appropriate databases. 



■ AESCLTS link in the main menu bar can be 
used to display a list of all previous search 
results. 



■ Queries operate in parallel on each host. 
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■ From the main menu bar, click RESULTS to 
retrieve the results of previous queries. 



■ By changing the start and stop dates, queries 
performed between those dates can be 
viewed. 

■ If the query name is known, it can be entered 
in the CUJER ; MM All E field. 

■ If the USERID is known, it can be entered. 

■ When complete, a window displays with the 
matching queries. 
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XKEYSCQRE Process Data Flow 
'/ Processing Programs 
Query Processes 
Other Processes 
Cronjohs 
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■ Processing programs are the main processes that extract metadata from 
the traffic and then database the information in insert databases. 



PROGRAM 


DESCRIPTION 


fiI@_input_proc 


Scans for new input files, (before 
processing moves the file to the .tmp 
directory of the input directory specified) 


sotf_dist 


Listens for incoming SOTF sessions 


process_data_parent 


Processes all new files discovered by 
file_input_proc or soif_dist; optionally 
archives content and databases 
metadata. Parent process loads all 
dictionaries and starts up, then forks child 
processes which do the actual 

processing. 
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parent 



■ This process replaces processed ataO through process_dataX 

■ The “parent” process starts up and loads all the dictionaries, and then “forks” 
child processes which actually do the processing 

■ Parent acts similar to the xks_app_launcher, managing restarts for the 
children when they die 

■ When dictionaries are modified, parent reloads them and restarts the children 

■ 5! xks proc 5 will show an ‘ fi X/Y iJ number next to p rocess_d ata_pa re n t 

• This is the number of children currently running, over the number that 
should be running (based on the xks.config num_data_processors setting) 

• pdp will show up yellow anytime X != Y and green when everything is 
running normally 

• This means when you first (re)start pdp, it will show yellow while it is 
loading the dictionaries, because none of the actual child process_data s 
are running yet 

■ xks proc 5 will report extra or missing process_dataX with a PI D of 0 

• Can’t tell what PID missing process_data is suppose to have, because its 
managed by the parent now 
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Query processes are processes that search 
and submit all necessary tables for the 
analysts queries. 



PROG RAM 


DESCRIPTION 


query_dispatch 


Submits search jobs to search databases 
and propagates the status of the search 
and results back to the web server 


query_proc 


Searches through all the necessary tables 
for the analysts queries. 
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Other process which is run from the 
Application Launcher. 




■ Mil@rdl©r_pr®(g - polls the 

/export/data/xkeyscore/outputs/mailorder_working 

directory by default. Then renames and moves 
mailorder files to 

/export/data/xkeyscore/outputs/mailorder. 
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■ ^ h@ = m @t® J in g ester- streams metadata over 
socket. This process improves database 
performance. Instead of each xscore_proc 
writing to the database independently, they 
stream their metadata over socket to the 
rnetajng ester, which combines it by plugin 
and writes to the database. 

• Reduces the number of connections to MySQL and 
gives better control over table size. 
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■ [mpmi^seimir - this is the snap-reduce server 
for microplugins, which runs the "Reducer” 
portion of GENESIS vS microplugins. 



■ Rums outside the normal processing flow, and 
will not affect the rest of the system. 

■ It has a telnet port (5850) just like an 
xscore_proc. 
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■ ©@rr@lati®n_@@[M@r_© - in-memory 




map-reciuce server for correlation engine. 



■ Each machine has one correlation_server, and 
every process_data_parent connects to every 
correlation server 



• xscore_proc - 8GB by default 

• uses port 4321 
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■ 2fe = ©@oimms = @(§rv(§r - a more efficient way to 
communicate with hosts within and outside an 
XKS cluster (not currently implement) 



• Automatically handles configuration for talking 
between slaves, master and overlord at site 

• Configuration is needed to connect to the “peer” 
on the path towards, other sites 

• Comrns configuration lives in 
$XSCORE_DIR/config/comms/comms.config 

• Supports a "quality of service" which “fairly 
distributes available bandwidth to the services that 
are using comrns 
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■ sslEi ©ootiims @®rv@r 



• Altova and Peer rules have a “network 1 ’ parameter 
which the cornms systems uses to determine an 
“inside' and an “outside 3 ’ in proxies. 

• Gonirns system will only accept connections from 
address ranges it has been specifically configured 
to allow. 

• Every between 2 comrns servers connection 
should have: 

> ' bandwidth_rule ;: on each side, name doesn’t matter but 
both rules should usually have same bandwidth cap 

t> allow" rule on one side with a reciprocal “peer” rules on 
the other side 
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■ sslEi ©ootiims @®rv@r 



• Example: If we have a site named “US-123 3 
connecting to xks-cemtral over a 1Mbps link, 

US-123 ;, s config would be: 

bandwidth [wo rid] = 1Mbps 

peer[GG] = address=xks-central.corp„nsa. ic.gov, port=2412, 
bandwidth=world, nettwork=external 

And xks-central would have: 

bandwidth[us123] = 1Mbps 

allow[00] = address=xkey-mast@r.us1 23, bandwidth=us123, 
network= interna I 
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■ Other process which is run from the 
Application Launcher. 

■ 00 Id] - rescans content against fingerprints 
when a user clicks to view the content of a 
session. 

■ t©omi©a]t.@h - web server used to host XKS 
G II I 

■ s@tft©dH;M@(g[rv(§r - downloads sessions 

• Gets called from the GUId process 

• Works with any downloaded traffic that is SO' I . 
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Other process which is run from the 
Application Launcher. 

rts = s©r¥@r_sti]ts - sends to 
xks_system_monitor on Master and generates 
stats about the server itself. 

• CPU usage, memory usage, disk space, disk I/O, 
network traffic, etc. 

• Stats are fed to xks_system_monitor and the 
system monitor does magic with them. 
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■ ^[k@ = s^st@mj-in]©not(9)r - collects stats 
messages from all over the system (front-end 
and back-end and the server itself) and 
summarizes them for forwarding. Optionally it 
can database stats locally. 
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XKEYSCORE uses a number of cron jobs to 
perform tasks. 



1 CKIOIMUOIS 


DESCRIPTION 


age_off_new.php 


Ages off metadata and 
content when the disk is 
near capacity, or when 
thresholds have been met. 


xks update_dictionaries 


Pulls updates from various 
sources. 


xks rsync push_config 


Copies the 
/opt/xkeyscore/config 
directory to the slaves. 


rwc_post_to_pub. py 


Once an hour kicks off an 
update request 
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Crontulb) is the program used to install, 
uninstall or list the tables used to drive the 
cron daemon. 



■ The crontab consists of 

• age_off_new.php 

• xks update_dictionaries 

• xks rsync puslijconfig 

• rwc_post_to_pub.py 
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dfifj [m@M.php 




• Options: 

> -debug : extra debug statements in the output 

> -info : extra info statements in the output 

> -task_db : explicitly state that the machine is a task host 

> -web_db : explicitly state that the machine is a web host 

> -nosleep : use if you want to run now 

• This process ages off tables and archived data 
based on the settings in the A§.<s®nfigj file and 
the percentage of disk space used. 
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■ sfe dli<gti®nari(§g 




• This process pulls the necessarily files from 
various sources to update the dictionary. 

• Configure /opt/xkeyscore/config/xks.config 

► # (dictionaries] 
dictionaryfO] = type=royale, \ 
src=sftp://tssi_fvey:tssi_fvey@ 
x ks - co nt ro l/h o m e/ts s i_f vey/x ks_d i ct_u pdate.tar.gz, \ 
dest=update/xks_dict_update.tar.gz, \ 
action[0]=“cd 

$XSCORE_DIR/config/dictionaries/update;$XSCOF!E_DIR/coof0g/ 
dictionari@s/update/dup_instiall.pl >/dev/null 2>&1“ 
dictionary[1] = type=cadence 
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■ sfe rswoKg pusth ©©nfog 




• Transfers Master configurations to its slaves. 



• Excludes dot files, “httpd/logs'”, 
loadserver/packag@s :i: , "httpd/Iog" 

• force: option to xks to force push_config when not 
on the master 
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■ rw© = [p@sft_t© = p)u[Q)o[p)^ 

• The automatic starProc process is as follows: 



> Hour 1 : master asks whoever (say xks-control) for an 
update, gets the rpm, installs it, there is much rejoicing. 
The slaves asks the master for the rpm at the same time 
the master asks xks-control, but obviously the master 
doesn’t have it, so nothing happens. 

> Hour 2: everyone asks for an update again, this time the 
master has the rpm, the slaves download it and install 
and there is much rejoicing. 

> The rpm is installed and process_data_parent’s are 
restarted as soon as the rpm is downloaded on a given 
machine. 
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What is a DeepDive? 

Why DeepDive? 

What does a DeepDive look like? 
Front-End Processes 



V xFIP 

V Promoter 
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XKEYSCORE packet processing solution 

• XKEYSCORE’s software handles all packet processing 

• No upfront filtering prior to XKEYSCORE 

• XKEYSCORE “promoter” tries to promote richest/most 
interesting traffic 

> All Strong Selectors 

> Full take ASDF (User Activity metadata) 

> Subset of GENESIS signatures 

■ List managed by XKEYSCORE team in concert with 
collection managers and site engineers 

• 20% - 30% of site traffic is fully processed and can be 
found via XKEYSCORE search 

> typically does not include unknown or uninteresting 
protocols 
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Access to most relevant DNI data supporting SigDev and 
collection missions. Enables new mission capabilities (e.gl 
Correlation) 



Session promotion can be synchronized and managed based on 
Genesis signatures, traditional tasking selectors and available 
resources 

> Provides better scaling 

> Drop unwanted data. Keep the rest and make decisions 
later and more accurately 

Better control of the processing space 

> Instantiate new mission capabilities and dataflows quickly 

> Troubleshooting and monitoring made easier 

Need access to “raw" packets to support new mission e.g., 
Cyber, Bulk Crypt: 

o Sessions can be displayed as Packet Bundles like Wireshark 
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What does a DEEPDIVE look like? 

• XKEYSCORE full-take session processor (Back End) 

• High speed packet ingest: an end-to-end solution 

• Intelligent filtering to vary the proportion of traffic retained 



; ^ 



Front End 



DEEPDIVE 

v 



Mettlesome 



Packet 

Splatter 



Promoter I Defrag 



Back End 



Dictionary 

Scanner 


r^Plugins^T 


Microplugins 1 




I Fingerprints I 



Metadata 



Content 



Packets 



c 




Partial Sessions 
1 Z 






Fuff Sessions 



c 
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Front 



What it's called 


What it does 


What it means 


Packet Splatter 


Ingests packets (from files, from the 
network, from a capture card) in a 
variety of formats. 


If it's a packet stream, it can probably 
be fed into a DEEPDIVE. 


xFip 


Fast reassembly of TCP/IPv4, 
UDP/IPv4 streams*, and TCP/IPv6 
and UDP/IPv6 streams*. 


DEEPDIVE sessionizes 


METTLESOME 


Reassembly of streams from less 
common protocol stacks. 


before making a keep/drop decision. 


Promoter 


Rule-based filtering of reassembled 
sessions, based on keyword, country 
code or appid/fingerprint. 


DEEPDIVE intelligently chooses the 
most useful traffic for retention. 


Defrag 


Fully rebuilds sessions** 


Enough content available to do full 
decoding/document descent at the 
Back End 
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■ Packet bundles 

• Preserves original packets and packet order 

• Preserves information that is lost during sessionization 

• Original pcap available in the XKS Viewer 

■ Packet API 

• Microplugins can iterate over raw packets 

• Microplugins can use information that is lost during 
sessionization 

> E.g. timestamps, flags, checksums 

■ Packet fingerprints 

• Fired based on observations xFip has made 

> E.g. large sequence gaps, TTL variation 
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Filters sessions prior to back end processing 

• keywords, regex, country code, appids* 

• SIGDEV: promotion rather than strong selection 

Set the focus of the back end 



traffic types of interest 
regions of interest 
legal/polScy constraints 



allow appid chat.* 



allow country_code FK 



block country_code US -US 



Set the width of the access aperture 

• promote 20% of 20 signals? 

• promote 100% of 4 signals? 

Set the length of data retention 

• promote 20% and keep for 3 days? 

• promote 30% and keep for 2 days? 
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v options 

V General Commands 
^ Services 

V Actions 
Options 
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Usage: xks [options] <command> 

' (ry 3 xks help <name>’ to get help on a specific 
service or action 

General commands: 



► services - list available services 

► actions - list available actions 

► dependencies [invert] - show service dependencies 

► help [items] - print help on services or actions 



• Services (specify one or more service names or 
‘all’): 



► start <services> 

► stop <services> 

► restart <services> 

► status <services> 

► setup <services> 



- start the specified services 

- stop the specified services 

- restart the specified services 

- print the status of the specified services 

- setup/configure/fix the current xks install 
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► accounts_report 

► add_admin 

► change_db_password 

► cluster 

* compile_genesis 

► disk_check 

► ext4_format 

► ext4_upgrade 
contents 

► fetch 

► forceregister 

► info 

install_s!ave 

► local tagging 



- sends an email containing accounts usage to the 

specified users 

- sets up a local Linux user to administer XKS 

- changes the XKS database user’s password and 
updates all references to it 

- cluster actions 

- compiles GENESIS signatures 

- get raid and disk status 

-format $XSCORE_DATA_DI R partition and convert 
to extl filesystem 

- convert to ext4 filesystem while preserving 

of $XSCORE DA A DIR (no formatting) 

- fetch a remote file 

-force metadata table registration 

- show cluster information 

- install a slave machine in this cluster 

- checks and/or loads tagging file 
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► top 

► update_dictionari@s 

► updat@_gui_help 

► users 

► version 

► watchdog 
b workflow 



- display system performance 

- update all XKS dictionaries 

- update the 'help' pull downs in Gill 

- display the users currently logged into the Gill 

- show XKS version information 

- check and (re) start essential XKS processes. 

- manually submit a workflow 
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• Options: 



► -verbose : print extra information to the screen 

► -debug : used for debugging script problems 
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> ID 01 



MJi 



I (1) l S IX 



l nr 1 1 ■ OOI I On 1 



i I I ml 



l< I A 



4 I V. | 
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General Commands 



"ype: xks help services 

> This will list all available services 




■ first - initialization service that runs before all others 

■ virus_scaDTiner- sets up virus scanner, assuming tarballs are 
present. 

■ PSpcI - enables ftp on the master if mailorder is enabled 

■ d is tee - sets up distributed compiler service 

■ slash jp roc - setup optimal / proc parameters 

■ myroeem - handles installation and configuration lOGigE 
network cards 

■ hem® - sets up the home directory for the xks user account 
□ gee - check there is a working compiler on the system 

■ upgrade - updates configuration files when upgrading to a 
new version of xks 

■ bash re - sets up bash environment variables 

■ beacon - sets up xks monitoring beacon based on xks.config 

■ tt- checks connectivity to TRAFFICTHIEF server 
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> ID 01 



MJi 



1 QQ l i LX 



infill (jm I HO 1 



1 1 I ml 



l< I A 



4 k V. | 
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General Commands 



"ype: xks help services 

> This will list all available services 




■ sendmail - configures sendmail for use with xks 

■ r©le_files - this service installs role-specific files 

■ ossu® - sets up the DoD mandatory login warnings 

■ r®^sil@_with_eh@@s@ - setups automatic updates 

■ ntpd - configure ntp based on xks.config 

■ linCc_s(LiQTuma[ry - sets up xks link summary GUI 

■ mfsd - sets up xks-specific nfs mounts 

■ s©ir¥@r_©©Dis - sets up server certificates for SSL applications 

■ ®p©n©W]e® - installs and configures OpenOffice for use in the 
xks GUI 

■ init_d - sets up the xks init.d services 

■ r@s©lv®r - sets up resolver config 

■ php - sets up PHP related stuff. Except php.ini 

■ httpd - sets up xks-specific httpd configuration 
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General Commands 



"ype: xks help services 

> '"his will list all available services 




■ www- sets up GUI configuration files 

■ ¥®ip- sets up voip processing 

■ ©r®md - ensures xks can use cron and sets up xks cron jobs 

■ sshd - configures the secure shell service for use with xks 

■ IScens® - checks for a valid license file and if one isn’t found 
prints a message 

■ s^slog - configures the syslog service for use with xks 

• all xks processes log to /var/log/xks.log 

■ dictionaries - checks status of any configured dictionaries 

■ clust®r_di®ck - checks network connectivity across the 
cluster 

■ autofs - start, stop, restart automounts 

■ loadsecwer- start, stop, and setup loadserver 

■ director!®© - sets up directories used for xks 

■ auditd - no help available 
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■ Idjap - no help available 

■ mysqld - sets up the mysql server for use with xks 

■ disks - checks status of disk partition used by xks 

■ databases - maintains database scheme consistency 

■ locsi Masking - reapplies local tasking if necessary 

■ workflows - sets up xks default workflows 

■ catego^Mih rattle - overrides default category throttle settings based 
on overrides specified in xks.config 



■ @nrichmni@nt_tO[nrQcatt- sets up enrichment tomcat java application 



server 



■ [9lugin_@<gt[up - populate plugin database tables from xml files, appy 
default plugin config specified in xks.config, apply overrides from 
xks.config, regenerate plugin config files from database 

■ crdb - no help available 

■ toGTfocat - sets up tomcat java application server 

■ cl do Best ream - sets up clickstream service 
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■ fila_inpu t- sets up directories and database entries needed for file- 
based input 

■ - synchronizes the database (xs_task_db.age_off) with 
xks.config’s settings for content and metadata. The values in the 
database will be unconditionally overwritten with those found in 
xks.config 

■ db_conro©etovity - verifies connectivity to critical databases 
- pdf - sets up xpdf language packs 

■ ul_ag@_off- sets the maximum data retention time to a little over an 
hour in UL mode. 



■ m DIMS [Res [ponder - sets up m DNS Responder for use with SGT U 
input 

■ ap|p_l®unch@r- controls the xks app launcher, which is responsible 
for monitoring xks processes and starting/stopping them as 
commanded from the GUI 



■ prcc®ss@s_s©ti(Lip- configures xks processes based on specifications 
in xks.config 

■ eooums - sets up the XKS communications system configuration 

- a/rlva nrorfTQE 8 M ftsi/fiffiiF ISA, AUS , £ H? , fJ^+inn 
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General Commands 



"ype: xks help services 

> This will list all available services 




■ (andac© - handles all the installation and configuration for 
Endace Dag packet capture cards 

■ last - cleanup service that runs after all others 
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► xks start mysql 

• stop 

t> xks stop httpd 

• restart 



► xks restart nfs 



• status 

I xks status autofs 




setup 



>xks setup plugins 
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• As f®r©@_r@gister 




gr@p ^s®@r® I gr@p -w grep s 



[operJtlxksvrOl rim] $ xks force register 



Forced update on xks_meta_ingester 



• As rs^n© push_©@nfig -f®r©@ 

> Usage: xks rsync <options> 
push_config push_compiled|push_slaves[push] <src> 
<dest> 



As updlsit@_©]i®ti®nari®@ 

> Usage: xks update_dictionaries 
[test|check|print|force|help] 



• As wsrsi©rs 



[opergtlxksvrQl run] $ 
1 . 5 . 9-65 



> version. 



As inf© 



[ o pe r - yt i >: ks vrOl run] $ jh ks info 
3 _Ll_.ee : Timbe t line — tSV' 

S I C4A D : U -S F - 7 9 □ 

PDDG : IE 

X K S ve rsio n : 1.5.9— iS5 

Master : tlxksvrl] 1 

I"J i_ij[l :=' X cj. v ee := ; 13 

E n put : file. f sotf 

II o n IT ± q : managed s art d 



I n put 
C o nfi q 



TOP SECRET //SI II REL TO USA, AUS, CAN, GBR, NZL 



96 




>10 01 W 
1 00 l I oc 
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xks - Actions 



o i I o i 










[oper@tlxksvrDl run] 5 query s; & r~ v rr s 

3a 14s; 93n S 4w 2 012-12- OS 16: 013? : 2:2 



2: 3 0 a Os s 



On iDw 2 012 - 12-05 13 ? : 55 : Oi 2 



\tz. 1 k; s: ’ur rr Oi 4 :: q Oi 
2: 3 □ a 01 s- 



On Ow Z0 12:-12:-0!5i 17:55: D2 



1 JL k. s; v nr" Qi 5c q Oi 
22 Sa □ s 



4 n Ow 2 0 12- 12- 01S. 13? : 5.5 :: 02 



11 s^arOV’ :: qOi 
2: 3: □ a. Ois: 



On Ow 2 012 - 12-05 17 : 55 : 02 



tlxksvrOO :: q Oi 

225a □ s Sn 



Ow 2 012-12-05 IV : 55 0'2 



ft 3L !k: s is - O l 9 ; :: q Ol 

□ .a Ois: 17 3: n 



2 w 2: □ 12- 12- OS* IV : S B : 0 2 



rtlxksvrlO :: qO 

2: 3Cla Os; O n 



Ow 2: □ 12- 12 - O S 17 : 5,5 :: 02 



1 X xks it 11: q Ol 
230a Os 



On Ow 2012-12-05 17:55: 02 



tlx:ksv3rl2 :: q Ol 
225 a □ s 



4n Ow 2 01 12 — 12 — 05 17::55: 02 



a = awaiting dispatch^ s = s;e:rt t ^ o=new, vi=work.ing 

timestamp shows earliest submitted l=ai_i"t— . unfinished que :ry 
current time 2012-12-05 1 Si : 0:2: : Q 3 
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> tot otoooi 1 



** Q| 101010 
} 10! 




II III 



A 

M 1 j A 




[oper@tlxksvrl.ll run] $ xks proc 
GUI GUId 


qp 


query_proc 


cli 


c 1 i c k s t r e am s e r v ice. s h 


rmt 


r e gi st e rmetadat at able s 


cms 


c h e c k m ail o r d er site. p h p 


s2d 


sot f t od 12 4se r ve r 


csOO 


c o rrel at. i o n s e r v e r 0 


sab 


signal acquisition base 


ct p 


c a d e n c e t a s k i n q p r o c 


sal 


signal acquisition loopback 


enr 


e n r i c hm & nt = t o m c at . s h 


sd 


sotf dist 


file 


file input proc 


sst 


strong selector targeting 


mp 


mailorder proc 


tom 


tomcat . sh 


mpmr 


mpmr server 


KGS 


Kks comms server 


pd# 


process_data# 


xmi 


Hks_meta_ingeste r 


pdp 


p r o c e s s _d at. a_p are nt 


xsm 


Kks_£iystem_monito r 


qd 


qu e ry_di s pat ch 


XSS 


x k s _3 e r ve r _s t at s 


Run ' 


xks proc full T to show full 


listing 





tlxksvrOl GUI cli cms ctp enr file m; 

tlxksvr02 csOO 

tlxksvfO-3 csOO 

tlxksvrOl csOO 

tlxksvrOS csOO 

tlxksvrO 6 csOO 

tlxksvr07 csOO 

tlKksvrOS csOO 

tlxksvrO 9 csOO 

tlxksvrlO csOO 

tlxksvrll csOO 

tlxksvrl2 csOO 

tlxksvrlS csOO 

tlxksvr-14 csOO 

Process consistency check OK on all hosts 



ctp 


enr file mp 






qd qp 


rmt 


s2d sab: 4/ 4 






sst 


tom xcs 


xmi 


xsm xss 


csOO 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/5 


sd 


xcs 


xmi 


xss 


csOO 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


csOO 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/ 6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/ 6 


sd 


xcs 


xmi 


xss 


csOO 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/ 6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal: 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 


cs 0 0 


mpmr 


pdp 


4/4 


qp 


rmt 


sab: 4/ 4 


sal : 


6/6 


sd 


xcs 


xmi 


xss 
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Exa rr i 
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pr@<s full 




[ o pe r @t 1 xks v r U 1 


run] $ xks proc full 












app 


launcher s t at. us: R OH ( p i d 3 0 7 U 8 ) 












id 


ho st name 


program 


argume nts 


commanded 


actual 


pid 


14 


tlxksvrOl 


c a d e n c e _t a s k i n g_p roc 


— rnyfdi XYD — pddg IE — dig. . . 


RUM 


ROM 




31136 


4 


tlxksvrOl 


c h e c k, m a i 1 o r d e r s i t e .. p b p 




RUM 


RUM 




31019 


7 23 


tlxksvrOl 


c 1 i c k s t. r e am £ e r v ice. s h 




RUM 


ROM 




31053 


654 


tlxksvrOl 


e n r i c h m e nt - 1 o m c at. . s h 




RUM 


ROM 




312 00 


9 


tlxksvrO 1 


file i n p u t p r o c 




RUM 


ROM 




31104 


1 


tlxksvrOl 


GUXd 




RUM 


ROM 




9335 


5 48 


tlxksvrOl 


m a i 1 o r d a r p r o c 


--copy dir / export/ data/ xkey. . . 


RUM 


ROM 




31140 


8 


tlxksvrO l 


q u e r y d i s p at c h 




RUM 


ROM 




175 49 


3 


tlxksvrOl 


query proc 




RUM 


ROM 




309 65 


193 


tlxksvrOl 


r a g i s t e r m e t. a d at. a t a b 1 e s 


--loglAvel error 


RUM 


RUM 




31143 


709 


tlxksvrOl 


s i g n a 1 a c q u isif i o n b ase 


- f generic packet to bundle. . . 


RUM 


ROM;: 


4/4 


31236 


12 


tlxksvrOl 


s o t f t o d 1 2 4 s a rve r 




RUM 


ROM 




31108 


46! 


tlxksvrOl 


s t r o n g s e 1 e c t o r t a r g eti n g 




RUM 


ROM 




31145 


13 


tlxksvrOl 


tomcat ■ sh 




RUM 


ROM 




31124 


653 


tlxksvrOl 


x k s c o mm s s a r v a r 




RUM 


ROM 




31148 


5 


tlxksvrOl 


x k s in eta i n g e s t e r 




RUM 


ROM 




31051 


4 62 


tlxksvrOl 


x k s s e r v a r s t at. s 




RUM 


ROM 




31138 


11 


tlxksvrOl 


x k s s y s t e m in o n i t. o r 




RUM 


ROM 




139 06 


724 


1 1 xks v r 0 2 


c o r if e 1 at i, o n_s e r v e r _ 0 


— leglevel debug 


RUM 


ROM 




139 46 


3! 


1 1 xks v r 0 2 


mpmr server 




RUM 


ROM 




7150 


730 


1 1 xks v r 0 2 


p r o c e s s _d at. a_p a r e n t 


-max-mem 20 


RUM 


RUM: 


4/4 


22661 


3° 


1 1 xks v r 0 2 


query proc 




RUM 


RUM 




14754 


137 


1 1 xks v r 0 2 


r e gi s t. e r met. adat a tab 1 e s 


— loglevel error 


RUM 


ROM 




14994 


710 


tlxksvfQ2 


s i gnal_a cqu i s i t i o n_ba s e 


-f generic packet to bundle . . . 


RUM 


ROM: 


4/4 


149 96 


41 


1 1 xks v r 0 2 


s i gnal a cqu i sit ion loo pb ack 


-f packet aux.config -i loo... 


RUM 


ROM: 


6/S 


14990 


259 


1. 1 xks v r 0 2 


sotf_dist 




RUM 


ROM 




148 45 


679 


1 1 xks v r 0 2 


x k s co mm s s a r v a r 




RUM 


ROM 




14992 


38 


1 1 xks v r 0 2 


x k s _iil e t a _ i n g e s t e r 




RUM 


RUM 




14970 


473 


1 1 xks v r 0 2 


x k s s e r v e r s t at s 




RUM 


ROM 




149 88 




_ _____ 


. 










„ ___ 
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• sdks q]u@rf 




[oper@tlxksvrQl run] $ 


xks query 














id 


us e r 


type 


search 


start 


search 


stop 


duration 


status 


66250201 




http_parser 


00:00 


12/2/ 12 


00:00 


12/6/ 12 


00:00: 10 


ongoing 


66250183 




full log 


00:00 


12/3/12 


00:00 


12/6/12 


00:00:17 


qikjo i ng 


66250155 




full log 


00:00 


12/ 4/ 12 


00:00 


12/6/12 


00 : 00: 40 


ongoing 


66250127 




geo_info 


00:00 


11/30/ 12 


00:00 


12/6/ 12 


00:01:16 


ongoing 


66250052 




email addresses 


00:00 


11/21/12 


00:00 


12/6/12 


00 : 03: 36 


ongoing 


66249073 




full log 


22:00 


12/3/12 


21:59 


12/4/12 


00:11:31 


o n g o i n g 


66249660 




full_log 


00:00 


12/2/ 12 


00:00 


12/6/ 12 


00:18:57 


ongoing 


66244233 




category 


00:00 


11/5/12 


00:00 


12/6/12 


00:42: 17 


ongo i ng 


66244135 




full log 


00:00 


11/28/ 12 


00:00 


12/6/12 


00:44:30 


ongoing 


66244009 




htt.p_pa.rser 


00:00 


11/5/ 12 


00:00 


12/6/12 


00:48:49 


ongoing 


66243967 




http parser 


00:00 


11/5/12 


00:00 


12/6/12 


00:49:34 


ongoing 


66243855 




doc urne nt metadata 


00:00 


11/21/ 12 


00:00 


12/6/ 12 


00:50:48 


ongoing 


66243785 




correlation 


00:00 


11/5/12 


00:00 


12/6/12 


00:52: 46 


ongoing 


66243463 




correlation 


00:00 


11/21/ 12 


00:00 


12/6/12 


00:56:13 


ongoing 


66243071 




email addresses 


00:00 


11/1/12 


00 : 00 


12/6/12 


01:08:24 


ongoing 


66242973 




user_activity_exif 


00:00 


11/21/ 12 


00:00 


12/6/ 12 


01:12:03 


ongoing 


66242413 




http parser 


00:00 


11/28/12 


00:00 


12/6/12 


01:26: 32 


ongoing 


66242315 




full log 


00:00 


12/4/12 


00 : 00 


12/6/12 


01:30:52 


ongoing 


There are 


18 queries 


in progress 
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2dk@ qjudirf dfetsiil 




[ o p s r @ 1 1 xk = 7 r 0 1 r u n ] l xk s q u e r y de tail i d - b 6 2510 6 5 

Query Summary 
Userid: 

T y p e : era a i 1 _a dd r e s g es 

S s a rc'ning from 0 0 : 0 D 1 1/ 30/12 to 0 0 : 0 0 12/ 6/ 12 
Duration: 00:01:25 
Priority: 5 
Cancel: X(o) 

Max Results: 10000 
Max Time: 6030 

Query 5 Cl 
Name: smstev3_4 

Classification: S, I S/ SI, MSA NQFGRM, T3, HC3, 3 / SI, MUSCULAR, REL USA, IJSANOFGRH, SI, C, R 

Where: WHERE datetime >= T 2012-11-30 00: 00: 00 T AND datetime <= r 20 12-12-06 00 : 00: 00 1 AND email - T 1 AMD domain - T hotniail.com T 



Query Status 



host 


database 


stat us 


tlxksvrOl 


qO 


finished 


tlxksvr02 


q0 


ongoing 


tlxksvr03 


qO 


finished 


tlxksvrQ4 


qO 


finished 


tlxksvrOl 


qO 


finished 


tlxksvrOS 


qO 


finished 


tlxksvrOl 


qO 


finished 


tlxksvrOO 


qO 


finished 


tlxksvr09 


qO 


finished 


tlxksvrll 


qO 


finished 


tlxksvrl2 


qO 


finished 


tlxksvrl3 


qO 


finished 


tlxksvrl4 


qO 


finished 


tlxksvrlO 


qO 


finished 
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w Executables 

^nmysqls 

onall 

^xks onall 
xks monitor 
v sotf_stat 
xks top 

V Web Status 

Additional Monitoring 
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1001 t 001 1001 



iDi n» 



>101 OlOC 

O m irtir 
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System monitoring can be performed from the 
command line using the following executable 
commands: 

miysqls 
onall 
xks onall 
sotf_stat 
xks top 
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The bash shell script can be used to 

execute MySQL statements from in the 
/optZxkeyscorefbin.shells/sysadmin/mysqls 
directory. Mie most commonly used options in 
oTaysqls are: 

status - displays file-based input statistics. 

sp@®d - displays the total file based input 
processing rate :Wlbps) 

sp©®d1 - displays file-based input processing rate 
(Mbps), per input source. 

sp@@d2 - displays file-based input processing rate 
::Mbps) per xkeyscore processing server. 

©©unit - displays the count of input files in the 
new, working, error, and done states. 
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) ID 01 

1 00 1 I oc 






Bk* < 
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m HHItl 




sis ©nail ‘sslks mwsqjl status’ 




1 [oper JL x 3c s vrOl 

D O y O 'Ll W EH Til 1z t O 


run] $ x 1c s o n si JL 1 
execute 1 ‘ R s mysql 


xks raysql 
s t sit u s " 


S t Slit u s; T 

0 ra.s.JL J_ V [ y | n] y 


II _ - tlKksvrOl 


-- — — — 






status 


mysqld 
my s qj JL cl 


is running 






1 

J 

* 

i 

rt 

h 1 


xksvrOS 


— 






stat us: 


myEqld 
nays qld 


is it im ri n .i 11 q 






I -- - t JL 


^ k: s; v" rr O 3 


-- — 






II st atus; 


mysqld 
my s q JL t:I 


is ifunninq 






II ----- -- tl k ks v t- O 4 









status 


ruysql <d 
my s qJL<d 


i s running 






^ tlKksvrOS 








status 


mysqld 

mysqld 


is running 
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> ID Qt 
I OO 1 1 



(Y) | 



I 



O I 



nr 
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■ This script will monitor your front-end 
processes. 




■ Type: ssks Gic©nt®r or xls ow@nit®r h to 

receive the help menu 



XK EYE COE E -M e mi 

Command 






Name 


Desc r i pt i, o n 


c: 


con fig 


C o n 1 i git re t h i s lit i 1 i ty 


d 


dat af 1 o w al 1 


F ro nt E n d Datafl ov M e n it 


b 


dataflow be 


B a c k E aid Dat at 1 ov Me nu 


m f h 


menu 


View this menu 


a 


pa c: ke t s p 1 at t e r 


P ack e t Acq ui sit i o n [Fro ait E n d ) 


P 


p t o ce s s d at a 


Process Data [Back Said] 


q 


quit 


Quit /Exit 


s 


serve rs 


S e r ' v e r S t at s (CPU ' f I etc. ) 


t 


sotf input 


SOTF Input [Back End] 1 xfcs top 1 replacement 


f 


xf i p 


Sessio n i z at i o n I| 5? ro nt E aid ] 
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■ Type: 2sks DTra®nt@r f to receive xfip stats 



KKEYSCORE-S^ssionizat idii [Server view Press T 4 1 l 



C^ss-te not suiion 


R ar e ( 


M bpe > 


XiOi c 


s t 


T C PQus 1 ir-y 


Psre ( Pkr ) 


Co vi.nn. 


CEk:t5 


P unt 


? rAgmente% 


7DH11S1 S L^flOOOC 




0,00 




0.00 


0. 00 


O. 00 




O 


O. OO 


0 . OO 


7DH1152Q90<i OOOO 




0.00 




0.00 


□ . oo 


O. DO 




□ 


o. oo 


□ . OO 
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■ The s@tf_stat command is used to display the 
SO" a ' (streaming object transfer format) input 
statistics for an entire cluster. 



■ The statistics include total number of 
pr©<g@s@ = dlata : @ running on the cluster, 
session input rate (sessions/sec), total bytes 
input (Mbps), and total bytes output to 
process_data(s( (Mbps). 
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■ To execute the script: 



• Log on to the server and open a terminal window. 

• Type sotf_stai because the command is in the 
path 

• Type s to toggle the summary statistics view from 
total statistics to individual host statistics. 

• Type q to quit the program 
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'."he @©tf_@tat script lists the hostname, 
number of p recess jdata s currently running, 
Mbps, number of sessions, and number of 
bytes. 



XKEY SCORE SOTF Statistics 



Hostname 


S In 


§QA / #OC 


Kbps In 


Sess In 


Bytes In 


mhxkssvr02 


7 


4/4 


18.66 


410920147 


4818112974976 


mhxkssvrOS 


3 


4/4 


16.15 


410121822 


4783549865004 


mhxkssvr04 


3 


4/4 


16.65 


410444622 


4781320276992 


mhxkssvrOS 


3 


4/4 


15.79 


409831857 


4759939303920 


—PRC: 15/ 15 


Rate 


: 64.52 


Mbps Sessions: 1641,; 


358767 Bytes: 



Sess Q MaxBlk 



3289212772' 
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"he top script lists the hostname, Mbps 
sotf rate, number of processjdata's running, 
tSie % of CPU, and % of 10 wait. 



hostname 


sotf 


tprocs 

i j 


cpul iowait% 


mhxkssvrOl 


-0,00 


0 


0,53 0,02 


mhxkssvrOS 


21 ,08 


4 


12,88 7,94 


mhxkssvr03 


13,55 


4 


13,35 7,28 


mhxkssvi:04 


14,97 


4 


14,50 8,63 


mhxkssvrOS 


14,13 


4 


17,14 8,01 


TOTAL 


63,74 


16 


11,68 6,38 
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no 01 101 iwi i™ 

1001 t 001 10O1 

jb ■ 







[oper@mhxkssvrQ2 “•]$ xks tail 

Dec 5 13:27:29 mhxkssvrQ2 register_metadata_tables (13877 j : cregister 
st (automatic?) repair failed 

Dec 5 13:27:29 mhxkssvr02 register_metadata_tables (13877 j :: < register 
st (automatic?) repair failed 

Dec 5 13:27:29 mhxkssvrQ2 register_metadata_tables (13877 j :: cregister 
st (automatic?) repair failed 

Dec 5 13:27:59 mhxkssvr02 register_metadata_tables (13877 j : < register 
st (automatic?) repair failed 

Dec 5 13:27:59 mhxkssvr02 register_metadata_tables (13877 j : cregister 
st (automatic?) repair failed 

Dec 5 18:27:59 mhxkssvr02 register_metadata_tables (13877 j : < register 
st (automatic?) repair failed 

Dec 5 18:28:08 mhxkssvr02 sotfjdist [13986] : <sotf_dist_t> NOTICE: cu 

Dec 5 18:28:29 mhxkssvr02 register_metadata_tables (13877 j :: cregister 
st (automatic?) repair failed 

Dec 5 18:28:29 mhxkssvr02 registerjnetadata_tables (13877 j : cregister 
st (automatic?) repair failed 

Dec 5 13:28:29 mhxkssvr02 registerjnetadata_tables (13877 j : cregister 
st (automatic?) repair failed 
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> ID 
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put® 

If there are too many files in the directory: 

> file_input_proc may be running improperly or not at all. 
Verify that fiIe_input_proc is running from the command 
line type: 

■ ps -ef | grep file ] grep -v grep 

■ xks proc 

> The file_input_proc may need to be restarted. 

No new files in the directory: 

The directory may not be cross-mounted properly, 
if automounting is used. 
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l®np©\r\MmtmlA®yw>mrdm)fBq\l\© @r il 
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• If /export/data/xkeyscore/mysql/iO or il are filling and 
qO and/or ql maintains its size, 
r®g]i8t@r_m@tsidaitsi_talbl®s may not be working 
properly. 



> Restart process and watch the databases to see if it is 
transferring files or run the process by hand to 
troubleshoot further. 



• If /export/daia/xkeyscore/mysql/qO or ql is filling, the 
ag@_®ff_n®w.php script may be running improperly or 
not at all. 



► First run the command: ps -ef 



grep age 



If script isn’t running, try running it by hand. 

If script is running, then stop script and try running it by hand to 
see if there are any errors. 
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• sotfjdist is listening for connections 

• If connections have been made to the sotf_dist 

• If we are "backing up”- i.e., if sotfjdist is running 
but has no process_data‘s connected to it, it won’t 
be able to send data anywhere, so eventually its 
network receive queue will get large. 

► Ideally, the receive queue should always be 0. 
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parent running? 



• At least one proeessjdata must be running and 
synchronized with the sotfjdist for it to receive 
input. 

> If problems continue, run the sotf_dist in a terminal to 
further troubleshoot and identify error messages. 
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■ Symptom: A lot of @rr@rs ®r tjoo many 
©ir®rs display wh©n pshhrmiM th© 



rnunaMnan 



fi myspls status’: 

1. First try, mysqls cleanup, in a terminal 
window. 

2. Type mysqls status 

3. Type mysqi xs task db; to „og into MySQL 

database and use the xs_task_db database. 

4. Execute the following command: delete 

from tar files where status= v error ; 

5. Exit out of the MySQL database 

6. Type mysqls status 

I Hi lh§yllTi(feJp&%l error files. 
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■ The heart of the XKEYSCORE processing 
engine is the xscore_proc with related plugins. 

■ Input to the xscore_proc is either file-based 
and from an file_input_proc, or streaming from 

an sotfJnput_proc. 

■ After processing, the written metadata to the 
insert databases can be sent to a follow on 
system for additional processing. 
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How miray [pr©)©@ssjdsita’s should lb® 
rnonning on a host? 

• From the XKEYSCORE GUI: 

► Click ADHIKI » Processing > Computer [Resources 

> Determine how many process_data’s are configured to 
be running on the specified host. 
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■ H ©wi omaa^ ^@©@r@_pr®© s s ar@ a©tually 
ro on du i mi g ®n a h®st? 

Log onto the XKEYSCORE server and open a 



terminal window. 



Type ps -ef 



grep xscore 



grep -v managed 
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■ 2fe = app_laoj][ri](gh@r is run ing, but n@t 
startling pr®e@ss@s sp@©ifi@d in th@ 
C®mput@r [M(is®ur©(§s wind®w? 



• This may indicate that the xks_app_launcher is 
defunct. Use the kill command to kill the 
appjaundier and its related sub-processes: 

> ' ype pkill -f app 

■ If a PID is not being specified, use the [©[kill command. The -f 
option kills all of the sub-processes. 

i ‘/ype ps to look for the new xks_app_launcher process. 
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If, after performing the procedures, the 
xks_app_launch@r is still not starting applications: 

> In a terminal window, manually run the problem process 
to see if there are any error messages. 

> The xks_app_launcher on any host is dependent on the 
access of the xs_task_db.proc_resources database 
table on the master. Verify that the specified host can 
access the master’s database and /opt directory. 

> On the slave system type mysql xs_task_db 

-h <masterhostname> 

■ performs a remote MlySQL server login 
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"g test the xscore_proc, type: 



telnet <process host> <port number > 



Optional commands to assist trouble shooting are: 

@(br - prints the processing rate for the single 

xscore-proc. 

sh - displays dictionary hit statistics. 

@s - displays statistics on the internal plug-in 

processing rates. 

help - there are many commands and can be 

described in the help menu. 
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■ If the pr@©(g@@ = ^at® = p)ar(gnt continues to deny 
access through the command port, and input 
still has not started processing, check the 
input source. 

■ Run the process in a terminal window with the 
argument --l®gjl®v@l dtertug), to view debug 
messages. 

■ The command port also provides processing 
rates and statistics for troubleshooting 
performance issues, outages, and general 
administration issues. 
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I@n p® oUMlfits/^ (kewsc® r@/@ u tp u ts/owa i I © rd @ r 



■ If there are no new files in the MAILORDER 
directory, MAILORDER may not be working 
properly. Possible causes are that: 

• Files are being written to the wrong directory or it 
is not configured properly 

• Permissions on the MAILORDER directory will not 
allow MAILORDER to move files 
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Ctetry dilp'atch is the process that submits 
search ,'obs to search databases and 
propagates the status of the search and the 
results of the search back to the web server. 

After submitting a new query, Search Status 
window displays a summary listing query 
name, date and time submitted, number of 
databases complete, and number of results. 
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■ The query never moves to the finished state. 



• If a database outage or a comms outage occurs, 
results will not be reported from the single system. 
However, results from all other databases will 
return properly with the query results, but they will 
not appear in this state. 
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■ @u<§[ff j@lb statins is stusk in 
awsiti ng = d is foatsh . 



• If a status appears stuck in this state, the 
query_dispatch may not be running on the web 
server. To determine whether it is running: 

> "ype ps -ef | grep query_ 

• If the process is not running, restart it from the 
XKEYSCORE GUI or troubleshoot the 
xks_app_launcher. 
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Another cause of this scenario is that a query 
database may have hung up the query dispatch 
process. Check the progress of queries on the 
query database hosts by viewing the table 
sdb_query_ Jobs in the query database, which 
tracks the status of queries: 



> ' ype mysql qO 

> '-VP 6 s elect status , count { * ) from sdb_query_j obs 
where group by status; 



The select statement displays the current state of 
the queries on the query host. If many more 
queries appear in the new state when compared 
to other query databases, begin troubleshooting 
the problem queryjproc on the specified query 
database. 
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■ The query is in the sent state, but never 



appears in new. 



• After the queryjdlispaifcgh process disbatches the 
query, the status is moved to sont. A query moves 
to the n@w state when the query has been placed 
in the query processing queue on the quervhost. 



• If a query does not move to the cnew state in a 
reasonable amount of time, the connectivity of the 
database should be tested. 



TOP SECRET //SI II REL TO USA, AUS, CAN, GBR, NZL 



137 



) ID 01 
1001 l 




TOP SECRET // SI // REL TO USA, AUS, CAN, GBR, NZL 





• To check the progress of queries on the query 
database hosts, view the table sdb_queij Jobs in 
the query database, which tracks the status of 



queries: 



► Type raysql qO 



ype select status, count (*) from sdb_query_j obs 
where group by status; 



• The statement displays the current state of 
the queries on the query host. If many more 
queries appear in the n©w state when compared 
to other query databases, begin troubleshooting 
the problem query_proc on the specified query 
database. 
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■ Th@ qpsiry app@@rs in th@ n@w st®t@ 3 (but 

HGr finiftii. 



• query is in the new state, has been received by 
the query host and placed in a queue waiting to be 
processed . 

o Queries can become backlogged with a large 
number of queries waiting in the new state, 
though the query_pmc is processing the queries 
properly. It is hard to predict the time to work off a 
query backlog, but using the following select 
statement the status of queries for the current day 
can be checked for processing trends. 
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©u|ri|| ©©caplet® but th@r@ ®r@ n® results. 

• If queries complete, but no results are visible, 
verify that the date range of the query coincides 
with the collection date of the data. If using test 
data, test the query system by putting the start 
date range at a year or two older to assure it is not 
old test data. 

• Verify that query metadata is in the query 
database by checking the contents of the 
/export/daia/xkeyscore/mysql/{query_dIh}/ 
directory. 
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©y@ri@s ©@mpl@t@ and raetadata returns, 
feat Charts i§ n® ©®nt@nt. 

• Th© metadata in th© XKEYSCQRE viewer 
displays the host and directory path of the content 
file. Verify the content file exists using the Is -I 
command. Trace a dataflow issue if the file does 
not exists. If th© content file exists, confirm th© 
httpd daemon is started on all Slav© systems. To 
confirm the httpd daemon: 

• 1 . Type su - oper 

• 2. Type xks status httpd 

o 3. If the daemon is noton, type xks start 

httpd 
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■ To troubleshoot problems with metadata or 
content from a query, it will be necessary to 
retrieve the actual content, since recreating 
the problem is very difficult. This can be 
accomplished from the XKEYSCORE GUI. 
Click RESULTS and begin a search of the 
questionable queries. 
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